The present invention relates to computer security.
Conventional computing systems include a user mode for providing user interaction with the computing system. The computing system can include a number of applications that can be interacted with by a user using, for example, a user interface. The user can direct the application to perform various functions. For example, a user selection of a particular function in an application can result in a system call to an application programming interface (“API”) associated with the application in order to initiate a process for executing the selected function. The functions can be performed in a user portion (“user mode”) or can require access to a kernel portion (“kernel mode”), a more secure level of the computing system that includes the core operating system. A function running in the kernel mode can execute all instructions and have access to all the system resources, including physical memory, input/output ports and other physical resources. In contrast, a function running in the user mode can access a smaller subset of instructions and a virtual memory that belongs to the user space. For example, functions such as opening a file or saving a file to memory require access to the computing system hardware (e.g., a hard disk drive or other memory device), which is only accessible through the kernel mode. Thus, for example, an open file call typically progresses from a user mode API to a kernel mode API. The kernel mode API then provides secure access to the device allowing, for example, access to physical memory address locations for opening a file.
Typically, application programmers do not have access to the source code of a user mode API. However, programmers can design programs to extend the functionality of the user mode API, using for example known hooks associated with the API. Conventionally, the applications are installed using an API interception (or API hooking) technique. Calls to or from the user mode API can be intercepted by an API interception application and additional functions performed prior to returning the call to the intended destination path. For example, a third party application such as a security system can install an API hook to intercept particular system calls from applications in order to determine whether or not the process has been affected by malicious software, commonly referred to as malware, including spyware, Trojans, viruses and other harmful code. The security system can then allow the call to continue (e.g., to the kernel mode) if the call passes one or more security checks. Intercepted calls which fail the security check(s) can be terminated.
Malicious software can be designed to circumvent security systems based on API hooking. For example, the malicious software can be designed to bypass the hook of the security system by calling the kernel API directly using an interrupt call or by inserting a jump command into the call thread prior to the hook in order to bypass the security system API hook.